Skip to content

#184 - Parameterize the bench suite for Spark/IGX and add IGX Orin performance report#185

Draft
dleshchev wants to merge 4 commits into
mainfrom
bench/igx-spark-suite
Draft

#184 - Parameterize the bench suite for Spark/IGX and add IGX Orin performance report#185
dleshchev wants to merge 4 commits into
mainfrom
bench/igx-spark-suite

Conversation

@dleshchev

Copy link
Copy Markdown
Collaborator

What

Closes #184 (IGX Orin C++ benchmarking) and makes the DGX Spark performance suite portable.

Two parts:

  1. Parameterize the bench suite for Spark and IGX via platform profiles. The *_spark* sweep configs and run_spark_* / setup_spark_wire_loopback_netns.sh scripts were hardcoded for DGX Spark (GB10). They now select a target with BENCH_PLATFORM (default spark, so existing Spark usage is unchanged):

    • examples/bench_platform.sh sources examples/bench_platform_<P>.env and fills @VAR@ placeholders in the config templates.
    • bench_platform_{spark,igx}.env carry the per-platform values: GPU memory kind, num_bufs, core map, DPDK port BDFs, wire-loopback netdevs/RDMA devices, RoCE flow-control depth cap.
    • Spark-filled output is byte-identical to the previous hardcoded config; IGX validated end-to-end.
    • Also fixes two issues surfaced on IGX: socket per-pair port substitution required quotes that PyYAML strips (every pair collided on the base port), and the RoCE client config used socket_config.remote_addr, which the current schema rejects.
  2. Add the IGX Orin performance report (docs/benchmarks/performance-igx-orin.md), the counterpart to the DGX Spark page, produced by the same scripts with BENCH_PLATFORM=igx.

Key finding

The offload-heavy bulk paths (DPDK raw, RoCE ≥64 KB) hit the same ~96–98 Gb/s NIC/PCIe ceiling on both platforms. The CPU-bound paths (kernel sockets, small-message RoCE, small-packet DPDK) trail Spark in proportion to the per-core CPU gap — IGX's 1.97 GHz Cortex-A78AE cores vs the 3.9 GHz Cortex-X925 cluster the Spark bench pins onto. The single-pair 1 MiB TCP ratio (17.8 vs 31.6 Gb/s = 1.78×) tracks the clock ratio, confirming the wire is not the limit. On the discrete RTX 6000 Ada, kind: device gives real GPUDirect (NIC DMAs into VRAM).

Path IGX Orin DGX Spark
DPDK raw / GPUDirect (8 KB) 96.4 98.5
RoCE SEND (8 MB) 97.7 102.2
RoCE SEND (8 KB) 12.8 60.7
TCP (8 KB ×4 pairs) 15.8 97.2
UDP (8 KB ×4 pairs) 9.3 29.8

Draft status / follow-ups

  • Single rep (REPEATS=1) — the report carries a callout in place of error bars. A REPEATS=3 re-run is planned before un-drafting, which will also clean up the single-queue CPU-utilization sampling.
  • Optional later: rename the *_spark* files to platform-neutral names (touches the CI doc-ref check + AGENTS table), and a Spark hardware re-validation pass.

Docs sync

mkdocs.yml nav, README Documentation table, AGENTS.md docs layout, and a Spark→IGX cross-link all updated. scripts/check_doc_refs.py passes.

🤖 Generated with Claude Code

dleshchev and others added 4 commits June 15, 2026 14:48
…ofiles

The DGX Spark sweep configs/scripts were hardcoded for Spark (GB10), so
running the same suite on the IGX Orin devkit (discrete RTX 6000 Ada +
ConnectX-7) meant editing files in place. Make the suite select a target
platform with BENCH_PLATFORM (default spark) instead.

- examples/bench_platform.sh sources examples/bench_platform_<P>.env and
  fills @var@ placeholders in the sweep config templates.
- bench_platform_{spark,igx}.env carry the per-platform values: GPU memory
  kind, num_bufs, core map, DPDK port BDFs, wire-loopback netdevs/RDMA
  devices, and the RoCE flow-control depth cap.
- The three *_spark* sweep configs become platform-neutral templates; the
  run_spark_* scripts and setup_spark_wire_loopback_netns.sh source the
  helper and fill a concrete base before any generator/PyYAML parses it.

The key per-platform difference is GPU memory placement: Spark's unified
memory uses kind: host_pinned, while IGX's discrete GPU uses kind: device
for real GPUDirect (DPDK raw ~96 vs ~74 Gb/s, RoCE large-message ~96 vs
~38 Gb/s -- the NIC reads VRAM over the GPU<->NIC PXB path instead of CPU
DDR over PCIe). Spark-filled output is byte-identical to the previous
hardcoded config; IGX validated end-to-end.

Also fixed two run_spark_bench.sh issues surfaced on IGX: the socket
per-pair port substitution required quotes that gen_spark_netns_config.py
(PyYAML) strips, so every pair collided on the base port; and the RoCE
client config used socket_config.remote_addr, which the current schema
rejects (peer now lives in rdma_bench_client.server_address).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Denis Leshchev <dleshchev@nvidia.com>
Add docs/benchmarks/performance-igx-orin.md, the IGX Orin devkit counterpart
to the DGX Spark performance report, produced by the same sweep scripts with
BENCH_PLATFORM=igx. Single-rep (REPEATS=1) C++-loopback numbers for DPDK raw
(kind: device GPUDirect), RoCE SEND, TCP, and UDP, with an "Single-rep figures"
callout in lieu of error bars.

Headline: the offload-heavy bulk paths (DPDK raw, RoCE >=64 KB) hit the same
~96-98 Gb/s NIC/PCIe ceiling as Spark, while the CPU-bound paths trail Spark in
proportion to the per-core CPU gap -- IGX's 1.97 GHz Cortex-A78AE cores vs the
3.9 GHz Cortex-X925 cluster the Spark bench pins onto. The single-pair 1 MiB TCP
ratio (17.8 vs 31.6 = 1.78x) tracks the clock ratio, confirming the wire is not
the limit.

- scripts/plot_mq_payload_sweep.py: parameterize the chart title (optional 3rd
  arg) so it is not hardcoded "DGX Spark"; render docs/images/igx-mq-payload-sweep.svg.
- Wire the page into mkdocs.yml nav, the README Documentation table, and the
  AGENTS.md docs layout list; add a Spark->IGX cross-link on the Spark page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Denis Leshchev <dleshchev@nvidia.com>
Re-ran the full IGX Orin suite with REPEATS=3 (BENCH_PLATFORM=igx) and replaced
the single-rep figures with 3-rep means. The numbers confirm the single-rep
draft almost exactly (run-to-run spread <=0.4 Gb/s on DPDK/RoCE, <=0.3 on
sockets), so the narrative is unchanged. Notable corrections from the cleaner
data:

- RoCE server RX core is ~4% busy (idle), not ~76% -- the expected RoCE RC
  signature (HCA writes straight to memory), matching Spark. The single-rep
  draft mis-attributed a busy worker core.
- DPDK CPU-utilization table replaced with an honest note: per-core busy% is
  unreliable on the nohz_full/rcu_nocbs isolated cores (tickless /proc/stat
  accounting under-counts and shifts run-to-run at identical throughput), so we
  don't publish precise per-core figures -- the throughput-vs-Spark gap is the
  real CPU-bound evidence.
- Regenerated docs/images/igx-mq-payload-sweep.svg from rep-averaged data.

Methodology callout updated to REPEATS=3; Reproduce section exports REPEATS=3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Denis Leshchev <dleshchev@nvidia.com>
…t results

The IGX socket pairs were pinned to the isolated cores (9 10 11 0) that DPDK/RoCE
use for poll-mode busy-spin. That tuning is wrong for kernel sockets: the
nohz_full/rcu_nocbs cores defer timers/RCU and starve the syscall path, and core
0 is the single sink for all NIC RX softirqs. Re-pin SOCKET_PAIR_CORES to the
non-isolated cores (1 2 3 4) in the irqaffinity=0-8 pool and re-run TCP+UDP at
REPEATS=3.

Effect: small-message TCP now scales ~linearly (1000 B: 4.2/8.5/16.7 over 1/2/4
pairs vs 3.6/7.1/12.1 before) and UDP improves across the board (8 KB 4-pair
10.5 vs 9.3 Gb/s). Large-message TCP is essentially unchanged because it is
receive-side bound, not app-core bound.

Report update: refreshed the TCP/UDP tables and added the measured root cause for
the large-message TCP ceiling -- all NIC RX softirq serializes on a single core
(cpu0 runs ~70% in softirq, every other core sees no NET_RX), so the four flows'
copy-heavy receive paths share one softirq core. RSS/RPS is not spreading receive
work; lifting it would need RX-softirq spreading, not more app cores. Documented
the socket-vs-poll-mode core split in the System-under-test table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Denis Leshchev <dleshchev@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] Benchmark DAQIRI on IGX Orin (C++)

1 participant